Structure-preserving clone brush

ABSTRACT

The invention provides tools and techniques for clone brushing pixels in an image while accounting for inconsistencies in apparent depth and orientation within the image. The techniques do not require any depth information to be present in the image, and the data structure of the image is preserved. The techniques allow for color compensation between source and destination regions. A snapping technique is also provided to facilitate increased accuracy in selecting source and destination positions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/951,355, filed Dec. 6, 2007, now allowed, entitled ‘StructurePreserving Clone Brush’, which is a continuation of application Ser. No.10/601,842, filed Jun. 23, 2003, now issued as U.S. Pat. No. 7,327,374,entitled ‘Structure Preserving Clone Brush.’ This application alsoclaims priority from U.S. provisional patent application No. 60/466,628,entitled ‘Image-Based Modeling and Photo Editing’, filed Apr. 30, 2003.Each of these applications is incorporated by reference herein in itsentirety.

FIELD OF INVENTION

The invention relates generally to computer graphics. More specifically,the invention relates to a system and methods for editing images whilepreserving the data structure of the images.

BACKGROUND

One of the most powerful and widely-used tools in photo editing is theclone brush, also known as the “clone stamp.” The clone brush permitsinteractive copying and pasting from one region of an image to anothervia a brush interface. It is often used to remove undesirable portionsof an image, such as blemishes or distracting objects in the background,or to replicate parts of a photograph. The brush interface facilitatesfine user control, and interactively copying and pasting existing pixelsenables the user to maintain photo-realism even with drasticmodifications.

Despite its utility and versatility, the traditional clone brush suffersfrom several important limitations. First, only regions with similarorientation and distance with respect to the camera can be brushedeffectively. Perspective foreshortening present in most photographs andvarious shapes of the objects in the scene make it difficult to clonebrush effectively. Although the traditional clone brush works well, forregions of an image that do not have a strong geometric structure, suchas clouds or vegetation, many structured features are not amenable forthe traditional clone brush due to perspective foreshortening, such asbuildings or tiled floors.

Second, intensity variations due to existing lighting in the photographfurther limit the effectiveness of the traditional clone brush.Artifacts appear when the intensity of the source and destinationregions do not match. Only regions with similar intensities can be clonebrushed convincingly using the traditional clone brush.

Third, it may be difficult to manually align the source and destinationpositions using a traditional clone brush. Misalignment between sourceand destination features leads to noticeable artifacts at the limit ofthe clone-brushed region. A precise initialization of source anddestination points is often necessary, especially for images that havestructured features, such as edges. It is common for the user to undoand repeat the initialization step until the points are sufficientlyaccurate.

While numerous software packages (such as ADOBE PHOTOSHOP, by AdobeSystems Incorporated, of San Jose, Calif.) provide clone brushingcapabilities in an image-editing environment, none of these packagesadequately addresses the problems of perspective foreshortening, colorcorrection, or accurately aligning source and destination positions.

What is needed, therefore, is editing software which allows clonebrushing that accounts for perspective foreshortening and lightingdifferences within the image. A need further exists for editing softwarethat allows a user to accurately align source and destination positionsfor clone brushing.

SUMMARY OF THE INVENTION

The invention provides a structure-preserving clone brush in a purely 2Dphoto editing context, where no depth information is necessary. Thistool can easily be integrated into existing 2D photoediting systems,such as ADOBE PHOTOSHOP, to improve the traditional clone brush.

The invention provides at least three enhancements to improve the 2Dversion of the traditional clone brush. First, the invention allows theuser to correctly copy and paste pixels according to the perspectiveforeshortening present on planar surfaces of the input image. The userprovides perspective information by tracing parallel lines in the image.

Second, the invention provides a simple color-correction technique thatallows the user to clone brush seamlessly between parts of the imagethat have different lighting conditions, by using a multiplicativefactor to compensate for the intensity variation.

Furthermore, the invention provides a “snapping” feature that allows theuser to initialize the source and destination positions more precisely.The new “snapping” feature optimizes the initial destination point tomatch features of the initial source point, thus providing preciseinitial points.

In accordance with one aspect of the invention, a method is provided forclone-brushing in an image while accounting for perspective andorientation differences within the image. The method comprisesspecifying a world plane, providing source and destination positions,identifying a destination region relative to the destination position,determining a source region relative to the world plane andcorresponding to the destination region, transforming image informationof the source region relative to the world plane to image information ofthe destination region, and copying the transformed image information tothe destination region.

In some embodiments, the source region is determined via a homographydefined by the world plane.

In some embodiments, the world plane is specified by drawing two sets ofparallel lines.

In some embodiments, a bilinear interpolation is applied, to imageinformation for the source region relative to the world plane.

In some embodiments, a first color region is provided for the sourceregion, a second color region is provided for the destination region, acolor ratio is computed for the color regions, and the color ratio isapplied during the transformation of image information from the sourceregion to the destination region. In one version, the color ratio iscomputed using Gaussian-weighted averages of the first and second colorregions. In one version, the first color region is provided with respectto the world plane.

In some embodiments, a second world plane and a relative scale factorare specified, the source region is determined relative to the firstworld plane and corresponding to the destination region relative to thesecond world plane and the relative scale factor, and the imageinformation is transformed relative to the first world plane to imageinformation for the destination region relative to the second worldplane and the relative scale factor. In one version, the second worldplane is specified by drawing a pair of parallel lines. In one version,the relative scale factor is specified by drawing a line segment of unitlength relative to the first world plane and drawing a line segment ofunit length relative to the second world plane.

In accordance with another aspect of the invention, a clone-brushingmethod is provided that compensates for color variation between regionsof the image. The method comprises providing first and second colorsample regions, computing a color ratio between the color sampleregions, providing source and destination positions in the image,identifying a destination region relative to the destination position,determining a source region corresponding to the destination region,applying the color ratio to image information of the source region andtransforming the image information of the source region to imageinformation of the destination region, and copying the transformed imageinformation to the destination region. In some embodiments, the colorratio is computed using Gaussian-weighted averages of the color sampleregions.

In accordance with another aspect of the invention, a clone-brushingmethod is provided for snapping an initial destination position to amore accurate destination position relative to the source position. Themethod comprises providing a source and initial destination position inthe image, determining a snapped destination position, identifying adestination region in the image relative to the snapped destinationposition, determining a source region in the image corresponding to thedestination region, transforming image information of the source regionto image information of the destination region, and copying thetransformed image information to the destination region.

In some embodiments, the snapped destination is determined by searchinga collection of candidate destination positions. In one version, aquality metric is applied to the candidate destination positions and acandidate whose quality is similar to the quality of the source positionis determined to be the snapped destination position. In one version,the quality metric for a position is a Gaussian-weighted color averagefor a region surrounding the position. In one version, the qualitymetric compensates for regional color variation by applying a colorratio.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and further advantages of the invention may be betterunderstood by referring to the following description taken inconjunction with the accompanying drawings, in which:

FIG. 1 illustrates a typical application of a traditional clone brushtool;

FIG. 2 illustrates a relative translation and problems of perspectiveforeshortening inherent in the use of a traditional clone brush;

FIG. 3 illustrates the use of homographies to apply a relativetranslation in a defined world plane, in accordance with an embodimentof the invention;

FIG. 4 illustrates a two-plane scenario for a structure-preserving clonebrush in accordance with an embodiment of the invention, along with theproblems of perspective foreshortening, misorientation and miscolorationinherent in the use of a traditional clone brush;

FIG. 5 illustrates a homography that linearly transforms points in theimage plane to points in the world plane, in accordance with anembodiment of the invention;

FIG. 6 illustrates the determination of a vanishing line and focallength, as used in computing a planar transformation, in accordance withan embodiment of the invention;

FIG. 7 illustrates the source and destination world planes in a 2-planescenario, along with the determination of rotation angle and relativescale factor, in accordance with an embodiment of the invention;

FIG. 8 is a flow diagram which illustrates a structure-preserving clonebrush method, in accordance with one embodiment of the invention;

FIG. 9 illustrates color correction in a clone brush operation as usedin a 2-plane scenario, in accordance with an embodiment of theinvention;

FIG. 10 is a flow diagram which illustrates a color correcting clonebrush method, in accordance with an embodiment of the invention;

FIG. 11 illustrates the snapping of a destination position, inaccordance with an embodiment of the invention;

FIG. 12 is a flow diagram which illustrates a snapping method for aclone brush, in accordance with an embodiment of the invention;

FIG. 13 illustrates an exemplary application of a structure-preservingclone brush, in accordance with an embodiment of the invention.

DETAILED DESCRIPTION

To use a traditional clone brush, a user first selects a “source region”of the image and then paints over a “destination region” using a brushinterface that copy-pastes pixels from the source to the destination.The use of a traditional clone brush is shown in FIG. 1, where the clonebrush has been used to remove the partial image of a woman from theimage.

An important concept of the traditional clone brush is the relativetranslation. Before clone brushing starts, the user selects the initialsource and destination points, s and d respectively, to define therelative translation, T_(r), computed in the image plane. As the clonebrushing progresses, T_(r), determines the position of the source regionwith respect to the currently-being-brushed destination region. Theequation to determine the location of the source pixel, x_(s), relativeto the current destination pixel, x_(d), isx_(s)=T_(r)x_(d)  (1)The color of pixel x_(s) is then copied to the pixel x_(d).

A more general formulation of Equation (1) isx_(s)=M_(r)x_(d)  (2)The matrix M_(r), is a similarity transformation matrix, whereM_(r)=T_(r)R_(r)S_(r). T_(r) is the relative translation, R_(r) is therelative rotation, and S_(r) is the relative uniform scaling that wouldallow the user to clone brush rotated and scaled pixels from the sourceto the destination region in the image plane. This feature is notavailable in traditional photo-editing software.

An exemplary application of relative translation in a traditional clonebrush operation is seen in FIG. 2. The traditional clone brush is to beused in this example to remove the chess pieces 200 from the chessboard202 in the image 204. The user in this example has chosen a sourceposition s 206 and a destination position d 208. This defines therelative translation T_(r) 210. As the user moves the cursor relative tothe destination position d 208, the corresponding position relative tothe source position s 206 is determined by the relative translationT_(r). Pixels are copied to the destination area 212 from thecorresponding source area 214. As noted previously, one of the mainproblems in clone brushing stems from perspective foreshortening. Thisis seen in FIG. 2 b, where the copied source region 214 does not appearto fit correctly within the destination region 212.

One aspect of the invention corrects the perspective distortions onplanar and quasi-planar surfaces that are inherent in the use of thetraditional clone brush. The traditional clone brush, as noted above, isbased on relative translation in image space. In the case offoreshortened geometry, it is desirable for the translation to takeplace in world space, as shown in FIG. 3. In an embodiment of theinvention, this is accomplished by defining the world plane using ahomography—a linear 2D-to-2D mapping between the world and image planes.Once the homography is determined, points on the image plane can bemapped to the world plane, and vice versa. A preferred embodiment of theinvention does not rectify images—rather, only the transformations thatdetermine the perspective-correcting coordinates of source pixels givendestination pixel locations are computed. The pixel copy of source colorto the destination position then occurs in the image plane. In theexample of FIG. 3, destination position d 300 in the image plane ismapped to world plane destination position d′ 302 via homography H_(d)304. Relative translation T_(r) 306 then maps d′ 302 to world planesource position s′ 308. The inverse homography H_(s) ⁻¹ 310 then maps s′308 back to image plane source position s 312. The pixels at s 312 arethen copied to d 300.

It is useful to discuss two types of clone-brushing scenarios: 1-planeand 2-plane. The 1-plane scenario is when the source and destinationregions reside on the same world plane. For instance, to remove thechess pieces from the chessboard in FIG. 3, only a single homographyneed be specified; homographies H_(d) and H_(s) are identical, mappingfrom the image plane to the world plane. The 2-plane scenario is whenthe source and destination regions reside in different world planes,e.g., clone brushing from one side of a building to another. In thiscase, homographies for both source and destination world planes need tobe specified, and additional parameters are necessary to determine therelationship between the two planes. In FIG. 3, if the source s 312 anddestination d 300 were located on different world planes within theimage, then homography H_(d) 304 and inverse homography H_(s) ⁻¹ 310would not represent the same homography: H_(d) 304 would map from theimage plane to the world plane containing d 300, while H_(s) ⁻¹ 310would map from the world plane containing s 312 to the image plane.

An example of the 2-plane scenario is seen in FIG. 4, where thechessboard 400 is partially folded into two distinct halves: ahorizontal half 402 and a vertical half 404. If a traditional clonebrush is used to copy the paper 406 from the horizontal half 402 to thevertical half 404, the result will be similar to that of FIG. 4 b, wherethe copied area 408 requires perspective correction relative to theplanes defined by both the horizontal half 402 and the vertical half404. Additionally, color correction is required due to the lightingvariation between the horizontal half 402 and the vertical half 404.

As illustrated in FIG. 5, points on the world plane are projections ofpoints on the image plane, and this linear mapping between the twoplanes is defined through a homography, i.e. a 2D projectivetransformation. More specifically, the homography is represented by a3×3 homogeneous matrix H. Points on the image plane x 500 are mapped tothe points on the world plane x′ 502 as x′=Hx up to a scale factor,where x is a homogenous column 3-vector x=(x,y,1)^(T). Similarly, theinverse homography, H⁻¹, maps points from the world plane back to theimage plane, x=H⁻¹x′.

The homography matrix H can be decomposed into the followingtransformation matrices:H=MN.  (3)The matrix M is a similarity transformation, which is also known as themetric component of the homography,M=T(t _(x) ,t _(y))R(θ)S(s),  (4)which has four degrees of freedom: translation matrix T with translationvector t=(t_(x), t_(y), 1)^(T), rotation matrix R with angle θ, andscale matrix S with uniform scale factor s.

The second non-metric component, N, determines the rectified world-planecoordinates. The matrix N can be further decomposed into N=AP, where Ais an affine transformation;

$\begin{matrix}{{A = \begin{pmatrix}a & h & 0 \\0 & 1 & 0 \\0 & 0 & 1\end{pmatrix}},} & (5)\end{matrix}$which has two degrees of freedom: a determines the width-to-heightaspect ratio, and h determines the shear transformation along thex-axis.

Matrix P represents a “pure projective” transformation:

$\begin{matrix}{{P = \begin{pmatrix}1 & 0 & 0 \\0 & 1 & 0 \\l_{1} & l_{2} & 1\end{pmatrix}},} & (6)\end{matrix}$where vector 1=(l₁, l₂,1)^(T) is the vanishing line of the world plane,that is, the horizon. Parallel lines on the world plane intersect at avanishing point on the image plane, and these vanishing points lie in l.This is seen in FIG. 6, where points u 600 and v 602 are vanishingpoints for parallel lines 604 on the image plane 608.

Only N is necessary to metric rectify an image, i.e., correctlyrepresent angles and length ratios up to a scale factor in the worldplane. Matrix M does not play a role in image rectification, but isapplied to rotate, uniform scale, and translate the image in its worldplane.

Unlike the traditional clone brush, the perspective-correcting clonebrush of the present invention uses a relative transformation M_(r) thattakes place in the world-plane coordinates. Similar to Equation (2), thefollowing formulation is used in a preferred embodiment to compute x_(s)on the image plane:x_(s)=H_(s) ⁻¹M_(r)H_(d)x_(d),  (7)where H_(s) and H_(d) are the source and destination homographies,respectively. More intuitively, H_(d) maps x_(d) to the destinationworld plane, M_(r) applies the relative transformation to thecorresponding position on the source world plane, then H_(s) ⁻¹ inversemaps the point back to the source region in the image plane. As theclone brushing progresses, the color of pixel x_(d) is replaced by thecolor of the source pixel x_(s), on the image plane, which corrects theperspective foreshortening problem. In a preferred embodiment, the colorof the source pixel x_(s) on the image plane is bilinearly interpolatedbefore being copied to x_(d).

In using the formulation in Equation (7), the metric components of bothhomographies can be ignored, i.e., H=N. The non-metric part alonesufficiently defines the world plane for the perspective correction.Furthermore, the transformation matrix M_(r) applies the relativetranslation, rotation, and scale necessary to relate the two planes.

For the 1-plane scenario, the source and destination homographies arethe same, i.e., H_(s)=H_(d), so the user only needs to specify onehomography. The user draws two orthogonal pairs of parallel lines in theinput image to specify a homography. This can be seen in FIG. 3, whereparallel lines p′ and parallel lines p″ have been drawn. In mostarchitectural scenes, these visual cues are commonplace. Alternatively,other techniques could be used to determine the homography, such asthose described by Liebowitz and Zisserman in “Metric rectification forperspective images of planes,” in Proceedings of the Conference onComputer Vision and Pattern Recognition, 1998, and by Liebowitz et al.in “Creating architectural models from images,” Computer Graphics Forum18, 3 (September), pp. 39-50. For non-planar geometry, other techniquescould be used locally to infer orientation of the surface, such as thosedescribed by Zhang et al. in “Single view modeling of free-form scenes,”in IEEE Computer Vision and Pattern Recognition, 2001.

The two sets of parallel lines drawn by the user specify vanishingpoints u and v. As shown in FIG. 6, the vanishing line l 606 passesthrough u 600 and v 602, where 1=(l₁, l₂,1)^(T)=uxv. Once l isidentified, the matrix P is determined (Equation (6)).

The parameters a and h of matrix A (Equation (5)) are determined bysolving the following quadratic equation for the complex number I:(1+2c _(x) l ₁ +l ₁ ²(c _(x) ² +c _(y) ² +f ²))I ²+2(l ₂ c _(x) +l ₁ c_(y) +l ₁ l ₂(c _(x) ² +c _(y) ² +f ²))I+2l ₂ c _(y) +l ₁ c _(y) +l ₂²(c _(x) ² +c _(y) ² +f ²)+1=0,  (8)where I=α−iβ, α=1/β, and s=−∝/β. The principal point 610, c=(c_(x),c_(y)), is assumed to be in the center of the image 608. The onlyunknown parameter is then the focal length f. The length of the linesegment f=oc 612 is determined as follows:

$\begin{matrix}{f = {\overset{\_}{oc} = \sqrt{{\overset{\_}{op}}^{2} - {\overset{\_}{cp}}^{2}}}} & (9)\end{matrix}$Point p 614 is the orthogonal projection of c 610 onto the vanishingline l 606 in image-plane coordinates. The length of line segment

$\begin{matrix}{\overset{\_}{op} = {\sqrt{\overset{\_}{up} \cdot \overset{\_}{pv}}.}} & (10)\end{matrix}$

Specifying two orthogonal sets of lines defines the non-metric part ofthe homography, N=AP. Therefore, the homographies H_(s)=H_(d)=N. In oneembodiment of the invention, the user then defines the relativetranslation, T_(r), by specifying the initial source and destinationpoints, as is the case with the traditional clone brush. The matrixT_(r) is now computed in world-plane coordinates, i.e. from t_(r)=s′−d′,where s′=H_(s)s and d′=H_(d)d, as discussed above with reference to FIG.3. Matrices R_(r) and S_(r) are irrelevant in this case, sinceH_(s)=H_(d), meaning their values do not affect the clone brushing. As aresult, M_(r)=T_(r), and Equation (7) is complete for the 1-planescenario.

In accordance with an embodiment of the invention, the 2-plane scenariobegins similarly to the 1-plane scenario, in that the user first draws apair of orthogonal parallel lines for both source and destinationplanes. These determine the non-metric components and, therefore, therespective homographies, H_(s)=N_(s) and H_(d)=N_(d).

Determining M_(r) for the 2-plan scenario is more complex since matricesR_(r) and S_(r) also need to be specified. To compute the rotation anglefor R_(r)(θ), the angle between the horizontal axis of the source andthe destination in the world plane is determined. A horizontal axis iscomputed by u′=Hu, where u′ represents the direction of the vanishingpoint u at infinity in world-plane coordinates. With reference to FIG.7, the destination plane 700 in the image is mostly vertical, and theworld plane for the destination is given by u 702 and v 704. The sourceplane 706 in the image is mostly horizontal, and the world plane for thesource is given by u 708 and v 710. The destination world plane 712 isshown in FIG. 7 b, where u′=H_(d)u, and v′=H_(d)v. The source worldplane 714 is shown in FIG. 7 c, where by u′=H_(s)u, and v′=H_(s)v. Therotation angle θ is defined as the angle between the source horizontalaxis (u′-axis 716) and the destination horizontal axis (u′-axis 718) inthe world plane. In one embodiment, the user specifies these axes whiletracing parallel lines for the homography, so no additional user inputis necessary. The first set of parallel lines drawn is the u-axis, andthe second set is the v-axis. Other methods of interactive specificationare also feasible.

In one embodiment, the relative scale factor, S_(r)(s), between thesource and the destination planes (Equation (5)) is determined via asimple interface. The user draws two line segments to be of equal lengthin the world space, and the lengths of the line segments l_(s) and l_(d)are computed in the respective source and destination world planes. Thisis shown in FIG. 7 a, where the user has drawn line segments l_(s) 720and l_(d) 722 in respective source plane 706 and destination plane 704.Line segments l_(s) 720 and l_(d) 722 are each three chess-squares inlength, and thus should represent an equal length in world space. Aratio is then determined that defines the relative scale factor,s=l_(s)/l_(d). Determining T_(r) is the same as in the 1-plane scenario.All the parameters in Equation (7) are now available, whereM_(r)=T_(r)(t_(x), t_(y))R(θ)S(s).

FIG. 8 is a flowchart illustrating a structure-preserving clone brushingmethod in one or two image planes, in accordance with an embodiment ofthe invention. The method begins by specifying a world plane in aspecification step 800. This is typically performed by drawing two setsof parallel lines, as described above. The user decides if thestructure-preserving clone brushing operation will take place in oneplane, or in two planes in a decision step 802. If two planes are to beused, a second world plane is specified in a specification step 804.Specification step 804 is typically performed by drawing two sets ofparallel lines. Unit lengths for both world planes are also specified ina length specification step 805.

The process continues by selecting a source position in a sourceselection step 806, and a destination position in a destinationselection step 808. Using the plane information and the unit lengths, ametric component and homographies between the world planes and the imageplane are computed in a computation step 810. If only one world plane isused, the homographies H_(s) and H_(d) are the same. The user thenselects a destination region, typically by moving a cursor over theimage, in a destination region selection step 812. The correspondingsource region is obtained by applying the metric component and thehomographies in an application step 814. The corresponding source regionis then copied to the destination region in a copying step 816.

Another common problem in traditional clone brushing occurs due tointensity variations in the input image. Even if the materials ortextures in the scene are the same, existing lighting conditionsinfluence their color. Although the clone-brushed chessboard patternsare aligned, the intensity variations make the clone brushed regionappear out of place.

In an embodiment of the invention, a triple of color-correction factorsis computed—the red, green and blue color ratios from the source to thedestination regions—which compensates for the intensity differences. AGaussian-weighted average between sample regions of the source anddestination images is used in computing the ratios. In a preferredembodiment, the ratios are computed during the initialization step (forexample, when the source and destination positions are chosen) accordingto the current brush size. The user may alternatively select sampleregions by drawing line segments that determine the position and radius.Other techniques of selecting sample color regions may be used.

The color correction method is described generally with reference toFIG. 9, where the user has defined source sample color region c_(s) 900and destination sample color region c_(d) 902. The ratio of the averagecolors then serves as the correcting factor: each pixel component copiedfrom the source region is multiplied by c_(d)/c_(s), to compensate forthe color difference. The paper in FIG. 9 has been clone brushed withcolor-correction from the source 904 to the destination 906.

In one embodiment of the invention, perspective-correction is used asdescribed above, and the color correction factors are computed in theworld plane using the homography matrices. The radius of thecolor-correction sample region then specifies an area in the worldplane, rather than the image plane. The Gaussian weight is computed withrespect to the world plane distance. This is seen in FIG. 8, where thesource sample color region c_(s) 800 and destination sample color regionc_(d) 802 are ellipses of different size and orientation yet representan equivalent area in world space.

FIG. 10 is a flowchart illustrating a color-correcting method for use ina clone brushing operation, in accordance with an embodiment of theinvention. The method begins by specifying a first color sample regionin a specification step 1000, and specifying a second color sampleregion in a specification step 1002. The color ratio between the colorsample regions is then computed in a computation step 1004. The processcontinues by selecting a source position in a source selection step1006, and a destination position in a destination selection step 1008.The user then selects a destination region, typically by moving a cursorover the image, in a destination region selection step 1010. Thecorresponding source region is then determined in a source obtainingstep 1012. The color of the corresponding source region is transformedby applying the color ratio in a color correcting step 1014. Thetransformed source region is then copied to the destination region in acopying step 1016.

FIG. 11 illustrates a clone-brush snapping tool, as used in anembodiment of the invention. A common problem in traditional clonebrushing occurs during the initialization step, when the user manuallyselects the initial source and destination points to specify therelative translation. Precise selection of these points is necessary,especially for images that have structured features, to avoidmisalignment and noticeable artifacts around the seams of the clonebrushed. In one embodiment, the present invention provides a method forsearching for a more accurate destination point that is most similar tothe initial source point. The user only needs to approximately clickaround the general area, and the closest matching point to the initialsource point is automatically computed.

The method, as used in a preferred embodiment, is described withreference to FIG. 11. The user initially chooses a source position s1100 and a destination position d 1102. A local region R_(d) 1104 aroundthe Initial destination point d 1102 is searched for the closestmatching point to s 1100. Each candidate destination point x_(d) in thelocal region R_(d) 1104 is compared to the source s 1100 by computing acolor average for a local window centered around candidate x_(d) and acolor average for a local window centered around s 1100. The coloraverage preferably is computed using a Gaussian-weighted color average.The color averages are compared, and the point x_(d)εR_(d) whose localwindow color average is closest to the color average for the localwindow around s is “snapped” and used as the new initial destinationpoint, d₀ 1106. This comparison may be achieved by computing aGaussian-weighted L² color difference between each candidate x_(d) ands, and then snapping to the candidate with the least L² colordifference. The Gaussian average is weighted from the center outwards,so pixels further from the center carry less weight in the average. TheGaussian-weighted average is then squared to obtain the L² value. Sincethe search space is small, a preferred embodiment uses an exhaustivesearch method, which finds d₀ almost instantaneously. In practice,allowing the local region R_(d) to span 20 pixels by 20 pixels, andallowing the comparison windows around x_(d) and s to span 16 pixels by16 pixels produces favorable results: it takes 0.15 seconds for anexhaustive search using a Pentium III, 850 MHz machine. In alternativeembodiments, other methods such as edge-detection may be used to comparecandidate destination points with the source position. Additionally,metrics other than Gaussian-weighted L² color difference may be used inthe comparison.

If the homography information is available, the Gaussian-weighted L²color difference is computed using Equation (7). By using thehomographies, the comparison windows centered around x_(d) and s mayhave different sizes and orientations in the image plane, but representequal sizes and orientations in their respective world planes.

If the color-correction factor has been computed, as described above,color correction is applied while computing the L² difference bymultiplying each pixel in the local window around s by the colorcorrection factor c_(d)/c_(s) prior to computing the Gaussian-weightedcolor average. Correcting color facilitates more accurate snapping whenthe source and destination positions are in differently lighted portionsof the image.

FIG. 12 is a flowchart illustrating a snapping method for use in a clonebrushing operation, in accordance with an embodiment of the invention.The method begins by selecting a source position in a source selectionstep 1200, and selecting an initial destination position in an initialdestination selection step 1202. A set of candidate destinations isdefined in a candidate definition step 1204. Candidate definition step1204 is typically performed by identifying a region of pixels locatednear the initial destination position. The Gaussian-weighted coloraverage is computed for a small region of pixels around the sourceposition in a source color averaging step 1206. For each candidatedestination position, the Gaussian-weighted color average is computedfor a small region of pixels around the candidate, and theGaussian-weighted color average is compared to the computed coloraverage for the source position, in a candidate color averaging andcomparison step 1208. The candidate whose color average is most similarto that of the source position is “snapped” as the new destinationposition in a snapping step 1210. Clone brushing then continues with theidentification of a destination region, typically by moving a cursorover the image, in a destination region selection step 1212. Thecorresponding source region is determined in a source obtaining step1214. The corresponding source region is then copied to the destinationregion in a copying step 1216.

FIG. 13 shows an exemplary application of a structure-preserving clonebrush, in accordance with an embodiment of the invention. The originalimage 1300 has been extended into a new image 1302 by copying portionsof the image 1300 while maintaining consistency with perspective. Theworld plane has been defined in this example by the sets of parallellines p′ 1304 and p″ 1306.

In some embodiments, the functionality of the systems and methodsdescribed above can be implemented as software on a general purposecomputer. In such an embodiment, the program can be written in any oneof a number of high-level languages, such as FORTRAN, PASCAL, C, C++,LISP, JAVA, or BASIC. Further, the program can be written in a script,macro, or functionality embedded in commercially available software,such as VISUAL BASIC. The program may also be implemented as a plug-infor commercially or otherwise available image editing software, such asADOBE PHOTOSHOP. Additionally, the software could be implemented in anassembly language directed to a microprocessor resident on a computer.For example, the software could be implemented in Intel 80×86 assemblylanguage if it were configured to run on an IBM PC or PC clone. Thesoftware can be embedded on an article of manufacture including, but notlimited to, a “computer-readable medium” such as a floppy disk, a harddisk, an optical disk, a magnetic tape, a PROM, an EPROM, or CD-ROM.

While the invention has been particularly shown and described withreference to specific embodiments, it should be understood by thoseskilled in the art that various changes in form and detail may be madetherein without departing from the spirit and scope of the invention asdefined by the appended claims. The scope of the invention is thusindicated by the appended claims and all changes which come within themeaning and range of equivalency of the claims are therefore intended tobe embraced.

1. A computer program product for clone-brushing in a 2D image, thecomputer program product comprising a nontransitory, tangible computerreadable medium having computer readable program code stored thereon,the computer readable program code including program code for: providinga destination position in the 2D image; identifying a destination regionin the 2D image relative to the destination position; determining asource region in the 2D image corresponding to the destination region;providing a first color sample region in the source region; providing asecond color sample region in the destination region; computing a colorratio between the first color sample region and the second color sampleregion; applying the color ratio to 2D image information of the sourceregion and transforming the 2D image information of the source region to2D image information of the destination region; and painting by copyingthe transformed 2D image information to the destination region.
 2. Thecomputer program product of claim 1, wherein the color ratio is computedusing Gaussian weighted averages of the first and second sample colorregions.
 3. The computer program product according to claim 1, whereinthe destination position is a snapped destination position.
 4. Thecomputer program product of claim 3, wherein determining a snappeddestination position comprises searching a collection of candidatedestination positions.
 5. A computer program product for clone-brushingin a 2D image, the computer program product comprising a nontransitory,tangible computer readable medium having computer readable program codestored thereon, the computer readable program code including programcode for: providing an initial destination position in the 2D image;determining a snapped destination position; identifying a destinationregion in the 2D image relative to the snapped destination position;determining a source region in the 2D image corresponding to thedestination region transforming 2D image information of the sourceregion to 2D image information of the destination region; and paintingby copying the transformed 2D image information to the destinationregion, wherein determining a snapped destination position comprisessearching a collection of candidate destination positions and whereindetermining a snapped destination position further comprises applying aquality metric to the source position, applying the quality metric tothe candidate destination positions, and determining a snappeddestination position from the collection of candidate destinationpositions whose quality is similar to the quality of the sourceposition.
 6. The computer program product of claim 5, wherein thequality metric is a Gaussian-weighted color average for a regionsurrounding the position.
 7. The computer program product of claim 5,wherein the quality metric compensates for regional color variation byapplying a color ratio.
 8. A system for clone-brushing in a 2D image,the system comprising: a computer comprising a processor, memory, and adisplay, the memory containing instructions that, when executed by theprocessor, cause the computer to: receive an input 2D image; interactwith a user to specify a first world plane in the 2D image; interactwith a user to provide a source position and a destination position inthe 2D image; interact with a user to identify a destination region inthe 2D image relative to the destination position; determine a sourceregion in the 2D image relative to the first world plane andcorresponding to the destination region including: define atransformation that maps the destination position relative to the firstworld plane to the source position relative to the first world planeusing a homography defined by the first world plane, and identify pixelsin the source region of the 2D image corresponding to pixels in thedestination region of the 2D image using the transformation and thehomography; transform 2D image information of the source region relativeto the first world plane to 2D image information of the destinationregion; and clone-brush by copying the transformed 2D image informationto the destination region, wherein the instructions that, when executedby the processor, cause the computer to: interact with a user to providea source position and a destination position in the 2D image, provide asnapped destination position.
 9. A system according to claim 8, whereinproviding a snapped destination position comprises searching acollection of candidate destination positions.